Capturing fine-grained details for video-based automation of suturing skills assessment.

OBJECTIVES: Manually-collected suturing technical skill scores are strong predictors of continence recovery after robotic radical prostatectomy. Herein, we automate suturing technical skill scoring through computer vision (CV) methods as a scalable method to provide feedback. METHODS: Twenty-two surgeons completed a suturing exercise three times on the Mimic™ Flex VR simulator. Instrument kinematic data (XYZ coordinates of each instrument and pose) were captured at 30 Hz. After standardized training, three human raters manually video segmented suturing task into four sub-stitch phases (Needle handling, Needle targeting, Needle driving, Needle withdrawal) and labeled the corresponding technical skill domains (Needle positioning, Needle entry, Needle driving, and Needle withdrawal). The CV framework extracted RGB features and optical flow frames using a pre-trained AlexNet. Additional CV strategies including auxiliary supervision (using kinematic data during training only) and attention mechanisms were implemented to improve performance. RESULTS: This study included data from 15 expert surgeons (median caseload 300 [IQR 165-750]) and 7 training surgeons (0 [IQR 0-8]). In all, 226 virtual sutures were captured. Automated assessments for Needle positioning performed best with the simplest approach (1 s video; AUC 0.749). Remaining skill domains exhibited improvements with the implementation of auxiliary supervision and attention mechanisms when deployed separately (AUC 0.604-0.794). All techniques combined produced the best performance, particularly for Needle driving and Needle withdrawal (AUC 0.959 and 0.879, respectively). CONCLUSIONS: This study demonstrated the best performance of automated suturing technical skills assessment to date using advanced CV techniques. Future work will determine if a "human in the loop" is necessary to verify surgeon evaluations.

Assessing the efficacy of dissection gestures in robotic surgery.

Our group previously defined a dissection gesture classification system that deconstructs robotic tissue dissection into its most elemental yet meaningful movements. The purpose of this study was to expand upon this framework by adding an assessment of gesture efficacy (ineffective, effective, or erroneous) and analyze dissection patterns between groups of surgeons of varying experience. We defined three possible gesture efficacies as ineffective (no meaningful effect on the tissue), effective (intended effect on the tissue), and erroneous (unintended disruption of the tissue). Novices (0 prior robotic cases), intermediates (1-99 cases), and experts (≥ 100 cases) completed a robotic dissection task in a dry-lab training environment. Video recordings were reviewed to classify each gesture and determine its efficacy, then dissection patterns between groups were analyzed. 23 participants completed the task, with 9 novices, 8 intermediates with median caseload 60 (IQR 41-80), and 6 experts with median caseload 525 (IQR 413-900). For gesture selection, we found increasing experience associated with increasing proportion of overall dissection gestures (p = 0.009) and decreasing proportion of retraction gestures (p = 0.009). For gesture efficacy, novices performed the greatest proportion of ineffective gestures (9.8%, p < 0.001), intermediates commit the greatest proportion of erroneous gestures (26.8%, p < 0.001), and the three groups performed similar proportions of overall effective gestures, though experts performed the greatest proportion of effective retraction gestures (85.6%, p < 0.001). Between groups of experience, we found significant differences in gesture selection and gesture efficacy. These relationships may provide insight into further improving surgical training.

Surgical gestures as a method to quantify surgical performance and predict patient outcomes.

How well a surgery is performed impacts a patient's outcomes; however, objective quantification of performance remains an unsolved challenge. Deconstructing a procedure into discrete instrument-tissue "gestures" is a emerging way to understand surgery. To establish this paradigm in a procedure where performance is the most important factor for patient outcomes, we identify 34,323 individual gestures performed in 80 nerve-sparing robot-assisted radical prostatectomies from two international medical centers. Gestures are classified into nine distinct dissection gestures (e.g., hot cut) and four supporting gestures (e.g., retraction). Our primary outcome is to identify factors impacting a patient's 1-year erectile function (EF) recovery after radical prostatectomy. We find that less use of hot cut and more use of peel/push are statistically associated with better chance of 1-year EF recovery. Our results also show interactions between surgeon experience and gesture types-similar gesture selection resulted in different EF recovery rates dependent on surgeon experience. To further validate this framework, two teams independently constructe distinct machine learning models using gesture sequences vs. traditional clinical features to predict 1-year EF. In both models, gesture sequences are able to better predict 1-year EF (Team 1: AUC 0.77, 95% CI 0.73-0.81; Team 2: AUC 0.68, 95% CI 0.66-0.70) than traditional clinical features (Team 1: AUC 0.69, 95% CI 0.65-0.73; Team 2: AUC 0.65, 95% CI 0.62-0.68). Our results suggest that gestures provide a granular method to objectively indicate surgical performance and outcomes. Application of this methodology to other surgeries may lead to discoveries on methods to improve surgery.

Using Real-time Feedback To Improve Surgical Performance on a Robotic Tissue Dissection Task.

Background: There is no standard for the feedback that an attending surgeon provides to a training surgeon, which may lead to variable outcomes in teaching cases. Objective: To create and administer standardized feedback to medical students in an attempt to improve performance and learning. Design setting and participants: A cohort of 45 medical students was recruited from a single medical school. Participants were randomly assigned to two groups. Both completed two rounds of a robotic surgical dissection task on a da Vinci Xi surgical system. The first round was the baseline assessment. In the second round, one group received feedback and the other served as the control (no feedback). Outcome measurements and statistical analysis: Video from each round was retrospectively reviewed by four blinded raters and given a total error tally (primary outcome) and a technical skills score (Global Evaluative Assessment of Robotic Surgery [GEARS]). Generalized linear models were used for statistical modeling. According to their initial performance, each participant was categorized as either an innate performer or an underperformer, depending on whether their error tally was above or below the median. Results and limitations: In round 2, the intervention group had a larger decrease in error rate than the control group, with a risk ratio (RR) of 1.51 (95% confidence interval [CI] 1.07-2.14; p = 0.02). The intervention group also had a greater increase in GEARS score in comparison to the control group, with a mean group difference of 2.15 (95% CI 0.81-3.49; p < 0.01). The interaction effect between innate performers versus underperformers and the intervention was statistically significant for the error rates, at F(1,38) = 5.16 (p = 0.03). Specifically, the intervention had a statistically significant effect on the error rate for underperformers (RR 2.23, 95% CI 1.37-3.62; p < 0.01) but not for innate performers (RR 1.03, 95% CI 0.63-1.68; p = 0.91). Conclusions: Real-time feedback improved performance globally compared to the control. The benefit of real-time feedback was stronger for underperformers than for trainees with innate skill. Patient summary: We found that real-time feedback during a training task using a surgical robot improved the performance of trainees when the task was repeated. This feedback approach could help in training doctors in robotic surgery.

Association of Suturing Technical Skill Assessment Scores Between Virtual Reality Simulation and Live Surgery.

Introduction: Robotic surgical performance, in particular suturing, has been linked to postoperative clinical outcomes. Before attempting live surgery, virtual reality (VR) simulators afford opportunities for training surgeons to learn fundamental technical skills. Herein, we evaluate the association of suturing technical skill assessments between VR simulation and live surgery, and functional clinical outcomes. Materials and Methods: Twenty surgeons completed a VR suturing exercise on the Mimic™ Flex VR simulator and the anterior vesicourethral anastomosis during robot-assisted radical prostatectomy (RARP). Three independent and blinded graders provided technical skill scores using a validated assessment tool. Correlations between VR and live scores were assessed by Spearman's correlation coefficients (ρ). In addition, 117 historic RARP cases from participating surgeons were extracted, and the association between VR technical skill scores and urinary continence recovery was assessed by a multilevel mixed-effects model. Results: A total of 20 (6 training and 14 expert) surgeons participated. Statistically significant correlations for scores provided between VR simulation and live surgery were found for overall and needle driving scores (ρ = 0.555, p = 0.011; ρ = 0.570, p = 0.009, respectively). A subanalysis performed on training surgeons found significant correlations for overall scores between VR simulation and live surgery (ρ = 0.828, p = 0.042). Expert cases with high VR needle driving scores had significantly greater continence recovery rates at 24 months after RARP (98.5% vs 84.9%, p = 0.028). Conclusions: Our study found significant correlations in technical scores between VR and live surgery, especially among training surgeons. In addition, we found that VR needle driving scores were associated with continence recovery after RARP. Our data support the association of skill assessments between VR simulation and live surgery and potential implications for clinical outcomes.

Tailored Feedback Based on Clinically Relevant Performance Metrics Expedites the Acquisition of Robotic Suturing Skills-An Unblinded Pilot Randomized Controlled Trial.

PURPOSE: Previously, we identified 8 objective suturing performance metrics highly predictive of urinary continence recovery after robotic-assisted radical prostatectomy. Here, we aimed to test the feasibility of providing tailored feedback based upon these clinically relevant metrics and explore the impact on the acquisition of robotic suturing skills. MATERIALS AND METHODS: Training surgeons were recruited and randomized to a feedback group or a control group. Both groups completed a baseline, midterm and final dry laboratory vesicourethral anastomosis (VUA) and underwent 4 intervening training sessions each, consisting of 3 suturing exercises. Eight performance metrics were recorded during each exercise: 4 automated performance metrics (derived from kinematic and system events data of the da Vinci® Robotic System) representing efficiency and console manipulation competency, and 4 suturing technical skill scores. The feedback group received tailored feedback (a visual diagram+verbal instructions+video examples) based on these metrics after each session. Generalized linear mixed model was used to compare metric improvement (Δ) from baseline to the midterm and final VUA. RESULTS: Twenty-three participants were randomized to the feedback group (11) or the control group (12). Demographic data and baseline VUA metrics were comparable between groups. The feedback group showed greater improvement than the control group in aggregate suturing scores at midterm (mean Δ feedback group 4.5 vs Δ control group 1.1) and final VUA (Δ feedback group 5.3 vs Δ control group 4.9). The feedback group also showed greater improvement in the majority of the included metrics at midterm and final VUA. CONCLUSIONS: Tailored feedback based on specific, clinically relevant performance metrics is feasible and may expedite the acquisition of robotic suturing skills.

Machine Learning to Delineate Surgeon and Clinical Factors That Anticipate Positive Surgical Margins After Robot-Assisted Radical Prostatectomy.

Purpose: Automated performance metrics (APMs), derived from instrument kinematic and systems events data during robotic surgery, are validated objective measures of surgeon performance. Our previous studies showed that APMs are strong outcome predictors of urinary continence after robot-assisted radical prostatectomy (RARP). We now use machine learning to investigate how surgeon performance (i.e., APMs) and clinical factors can predict positive surgical margins (PSMs) after RARP. Methods: We prospectively collected data of patients undergoing RARP at our institution from 2016 to 2019. Random Forest model predicted PSMs based on 15 clinical factors and 38 APMs from 11 standardized RARP steps. Out-of-bag Gini impurity index determined the top 10 variables of importance (VOI). APMs in the top 10 VOI were assessed for confounding effects by extracapsular extension (ECE) and pathologic T (pT) through Poisson regression with Generalized Estimating Equation. Results: 55/236 (23.3%) cases had PSMs. Of the 55 cases with PSMs, 9 (16.4%) were pT2 and 46 (83.6%), pT3. The full model, including clinical factors and APMs, achieved area under the curve (AUC) 0.74. When assessing clinical factors or APMs alone, the model achieved AUC 0.72 and 0.64, respectively. The strongest PSM predictors were ECE and pT stage, followed by APMs in specific steps. After adjusting for ECE and pT stage, most APMs remained as independent predictors of PSM. Conclusion: Using machine learning methods, we found that the strongest predictors of PSMs after RARP are nonmodifiable, disease-driven factors (ECE and pT). While APMs provide minimal additional insight into when PSMs may occur, they are nonetheless capable of independently predicting PSMs based on objective measures of surgeon performance.

Road to automating robotic suturing skills assessment: Battling mislabeling of the ground truth.

OBJECTIVE: To automate surgeon skills evaluation using robotic instrument kinematic data. Additionally, to implement an unsupervised mislabeling detection algorithm to identify potentially mislabeled samples that can be removed to improve model performance. METHODS: Video recordings and instrument kinematic data were derived from suturing exercises completed on the Mimic FlexVR robotic simulator. A structured human consensus-building process was developed to determine Robotic Anastomosis Competency Evaluation technical scores across 3 human graders. A 2-layer long short-term memory-based classification model used instrument kinematic data to automate suturing skills assessment. An unsupervised label analyzer (NoiseRank) was used to identify potential mislabeling of skills data. Performance of the long short-term memory model's technical skill score prediction was measured by best area under the curve over the training runs. NoiseRank outputted a ranked list of rated skills assessments based on likelihood of mislabeling. RESULTS: 22 surgeons performed 226 suturing attempts, which were broken down into 1,404 individual skill assessment points. Automation of needle entry angle, needle driving, and needle withdrawal technical skill scores performed better (area under the curve 0.698-0.705) than needle positioning (0.532) at baseline using all available data. Potential mislabels were subsequently identified by NoiseRank and removed, improving model performance across all domains (area under the curve 0.551-0.766). CONCLUSION: Using ground truth labels from human graders and robotic instrument kinematic data, machine learning models have automated assessment of detailed suturing technical skills with good performance. Further, an unsupervised mislabeling detection algorithm projected mislabeled data, allowing for their removal and subsequent improvement of model performance.

Survival Analysis Using Surgeon Skill Metrics and Patient Factors to Predict Urinary Continence Recovery After Robot-assisted Radical Prostatectomy.

BACKGROUND: It has been shown that metrics recorded for instrument kinematics during robotic surgery can predict urinary continence outcomes. OBJECTIVE: To evaluate the contributions of patient and treatment factors, surgeon efficiency metrics, and surgeon technical skill scores, especially for vesicourethral anastomosis (VUA), to models predicting urinary continence recovery following robot-assisted radical prostatectomy (RARP). DESIGN, SETTING, AND PARTICIPANTS: Automated performance metrics (APMs; instrument kinematics and system events) and patient data were collected for RARPs performed from July 2016 to December 2017. Robotic Anastomosis Competency Evaluation (RACE) scores during VUA were manually evaluated. Training datasets included: (1) patient factors; (2) summarized APMs (reported over RARP steps); (3) detailed APMs (reported over suturing phases of VUA); and (4) technical skills (RACE). Feature selection was used to compress the dimensionality of the inputs. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: The study outcome was urinary continence recovery, defined as use of 0 or 1 safety pads per day. Two predictive models (Cox proportional hazards [CoxPH] and deep learning survival analysis [DeepSurv]) were used. RESULTS AND LIMITATIONS: Of 115 patients undergoing RARP, 89 (77.4%) recovered their urinary continence and the median recovery time was 166 d (interquartile range [IQR] 82-337). VUAs were performed by 23 surgeons. The median RACE score was 28/30 (IQR 27-29). Among the individual datasets, technical skills (RACE) produced the best models (C index: CoxPH 0.695, DeepSurv: 0.708). Among summary APMs, posterior/anterior VUA yielded superior model performance over other RARP steps (C index 0.543-0.592). Among detailed APMs, metrics for needle driving yielded top-performing models (C index 0.614-0.655) over other suturing phases. DeepSurv models consistently outperformed CoxPH; both approaches performed best when provided with all the datasets. Limitations include feature selection, which may have excluded relevant information but prevented overfitting. CONCLUSIONS: Technical skills and "needle driving" APMs during VUA were most contributory. The best-performing model used synergistic data from all datasets. PATIENT SUMMARY: One of the steps in robot-assisted surgical removal of the prostate involves joining the bladder to the urethra. Detailed information on surgeon performance for this step improved the accuracy of predicting recovery of urinary continence among men undergoing this operation for prostate cancer.

Technical Skill Impacts the Success of Sequential Robotic Suturing Substeps.

Introduction: Robotic surgical performance, in particular suturing, has been associated with postoperative clinical outcomes. Suturing can be deconstructed into substep components (needle positioning, needle entry angle, needle driving, and needle withdrawal) allowing for the provision of more specific feedback while teaching suturing and more precision when evaluating suturing technical skill and prediction of clinical outcomes. This study evaluates if the technical skill required for particular substeps of the suturing process is associated with the execution of subsequent substeps in terms of technical skill, accuracy, and efficiency. Materials and Methods: Training and expert surgeons completed standardized sutures on the Mimic™ Flex virtual reality robotic simulator. Video recordings were deidentified, time annotated, and provided technical skill scores for each of the four suturing substeps. Hierarchical Poisson regression with generalized estimating equation was used to examine the association of technical skill rating categories between substeps. Results: Twenty-two surgeons completed 428 suturing attempts with 1669 individual technical skill assessments made. Technical skill scores between substeps of the suturing process were found to be significantly associated. When needle positioning was ideal, needle entry angle was associated with a significantly greater chance of being ideal (risk ratio [RR] = 1.12, p = 0.05). In addition, ideal needle entry angle and needle driving technical skill scores were each significantly associated with ideal needle withdrawal technical skill scores (RR = 1.27, p = 0.03; RR = 1.3, p = 0.03, respectively). Our study determined that ideal technical skill was associated with increased accuracy and efficiency of select substeps. Conclusions: Our study found significant associations in the technical skill required for completing substeps of suturing, demonstrating inter-relationships within the suturing process. Together with the known association between technical skill and clinical outcomes, training surgeons should focus on mastering not just the overall suturing process, but also each substep involved. Future machine learning efforts can better evaluate suturing, knowing that these inter-relationships exist.

The Relationship Between Technical Skills, Cognitive Workload, and Errors During Robotic Surgical Exercises.

Purpose: We attempt to understand the relationship between surgeon technical skills, cognitive workload, and errors during a simulated robotic dissection task. Materials and Methods: Participant surgeons performed a robotic surgery dissection exercise. Participants were grouped based on surgical experience. Technical skills were evaluated utilizing the validated Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. The dissection task was evaluated for errors during active dissection or passive retraction maneuvers. We quantified cognitive workload of surgeon participants as an index of cognitive activity (ICA), derived from task-evoked pupillary response metrics; ICA ranged 0 to 1, with 1 representing maximum ICA. Generalized estimating equation (GEE) was used for all modelings to establish relationships between surgeon technical skills, cognitive workload, and errors. Results: We found a strong association between technical skills as measured by multiple GEARS domains (depth perception, force sensitivity, and robotic control) and passive errors, with higher GEARS scores associated with a lower relative risk of errors (all p < 0.01). For novice surgeons, as average GEARS scores increased, the average estimated ICA decreased. In contrast, as average GEARS increased for expert surgeons, the average estimated ICA increased. When exhibiting optimal technical skill (maximal GEARS scores), novices and experts reached a similar range of ICA scores (ICA: 0.47 and 0.42, respectively). Conclusions: This study found that there is an optimal cognitive workload level for surgeons of all experience levels during our robotic surgical exercise. Select technical skill domains were strong predictors of errors. Future research will explore whether an ideal cognitive workload range truly optimizes surgical training and reduces surgical errors.

An Assessment Tool to Provide Targeted Feedback to Robotic Surgical Trainees: Development and Validation of the End-To-End Assessment of Suturing Expertise (EASE).

Purpose: To create a suturing skills assessment tool that comprehensively defines criteria around relevant sub-skills of suturing and to confirm its validity. Materials and Methods: 5 expert surgeons and an educational psychologist participated in a cognitive task analysis (CTA) to deconstruct robotic suturing into an exhaustive list of technical skill domains and sub-skill descriptions. Using the Delphi methodology, each CTA element was systematically reviewed by a multi-institutional panel of 16 surgical educators and implemented in the final product when content validity index (CVI) reached ≥0.80. In the subsequent validation phase, 3 blinded reviewers independently scored 8 training videos and 39 vesicourethral anastomoses (VUA) using EASE; 10 VUA were also scored using Robotic Anastomosis Competency Evaluation (RACE), a previously validated, but simplified suturing assessment tool. Inter-rater reliability was measured with intra-class correlation (ICC) for normally distributed values and prevalence-adjusted bias-adjusted Kappa (PABAK) for skewed distributions. Expert (≥100 prior robotic cases) and trainee (<100 cases) EASE scores from the non-training cases were compared using a generalized linear mixed model. Results: After two rounds of Delphi process, panelists agreed on 7 domains, 18 sub-skills, and 57 detailed sub-skill descriptions with CVI ≥ 0.80. Inter-rater reliability was moderately high (ICC median: 0.69, range: 0.51-0.97; PABAK: 0.77, 0.62-0.97). Multiple EASE sub-skill scores were able to distinguish surgeon experience. The Spearman's rho correlation between overall EASE and RACE scores was 0.635 (p=0.003). Conclusions: Through a rigorous CTA and Delphi process, we have developed EASE, whose suturing sub-skills can distinguish surgeon experience while maintaining rater reliability.

Virtual Reality vs Dry Laboratory Models: Comparing Automated Performance Metrics and Cognitive Workload During Robotic Simulation Training.

Background: This study compares surgical performance during analogous vesico-urethral anastomosis (VUA) tasks in two robotic training environments, virtual reality (VR) and dry laboratory (DL), to investigate transferability of skill assessment across the two platforms. Utilizing computer-generated performance metrics and pupillary data, we evaluated the two environments to distinguish surgical expertise and ultimately whether performance in the VR simulation correlates with performance in live robotic surgery in the DL. Materials and Methods: Experts (≥300 cases) and trainees (<300 cases) performed analogous VUAs during VR and DL sessions on a da Vinci robotic console following an Institutional Review Board (IRB) approved protocol (HS-16-00318). Twenty-two metrics were generated in each environment (kinematic metrics, tissue metrics, and biometrics). The DL included 18 previously validated automated performance metrics (APMs) (kinematics and event metrics) captured by an Intuitive system data recorder. In both settings, Tobii Pro Glasses 2 recorded the task-evoked pupillary response (reported as Index of Cognitive Activity [ICA]) to indicate cognitive workload, analyzed by EyeTracking cognitive workload software. Pearson correlation, Mann-Whitney, and independent t-tests were used for the comparative analyses. Results: Our study included six experts (median caseload 1300 [interquartile range 400-3000]) and 11 trainees (25 [0-250]). A total of 8/9 metrics directly comparable between VR and DL showed significant positive correlation (r ≥ 0.554, p ≤ 0.032); 5/22 VR metrics distinguished expertise, including task time (p = 0.031), clutch usage (p = 0.040), unnecessary needle piercing (p = 0.026), and suspected injury to the endopelvic fascia (p = 0.040). This contrasts with 14/22 APMs in DL (p ≤ 0.038), including linear velocities of all three instruments (p ≤ 0.038) and dominant-hand instrument wrist articulation (p = 0.013). Trainees experienced higher cognitive workload (ICA) in both environments when compared with experts (p < 0.036). Conclusions: Most performance metrics between VR and DL exhibited moderate to strong correlations, showing transferability of skills across the platforms. Comparing training environments, APMs during DL tasks are better able to distinguish expertise than VR-generated metrics.

Surgeon Automated Performance Metrics as Predictors of Early Urinary Continence Recovery After Robotic Radical Prostatectomy-A Prospective Bi-institutional Study.

BACKGROUND: During robotic surgeries, kinematic metrics objectively quantify surgeon performance. OBJECTIVE: To determine whether clinical factors confound the ability of surgeon performance metrics to anticipate urinary continence recovery after robot-assisted radical prostatectomies (RARPs). DESIGN SETTING AND PARTICIPANTS: Clinical data (patient characteristics, continence recovery, and treatment factors) and surgeon data from RARPs performed between July 2016 and November 2018 were prospectively collected. Surgeon data included 40 automated performance metrics (APMs) derived from robot systems (instrument kinematics and events) and summarized over each standardized RARP step. The data were collected from two high-volume robotic centers in the USA and Germany. Surgeons from both institutions performed RARPs. The inclusion criteria were consecutive RARPs having both clinical and surgeon data. INTERVENTION: RARP with curative intent to treat prostate cancer. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: The outcome was 3- and 6-mo urinary continence recovery status. Continence was defined as the use of zero or one safety pad per day. Random forest (SAS HPFOREST) was utilized. RESULTS AND LIMITATIONS: A total of 193 RARPs performed by 20 surgeons were included. Of the patients, 56.7% (102/180) and 73.3% (129/176) achieved urinary continence by 3 and 6 mo after RARP, respectively. The model anticipated continence recovery (area under the curve = 0.74, 95% confidence interval [CI] 0.66-0.81 for 3-mo, and area under the curve = 0.67, 95% CI 0.58-0.76 for 6 mo). Clinical factors, including pT stage, confounded APMs during prediction of continence recovery at 3 mo after RARP (Δß median -13.3%, interquartile range [-28.2% to -6.5%]). After adjusting for clinical factors, 11/20 (55%) top-ranking APMs remained significant and independent predictors (ie, velocity and wrist articulation during the vesicourethral anastomosis). Limitations included heterogeneity of surgeon/patient data between institutions, although it was accounted for during multivariate analysis. CONCLUSIONS: Clinical factors confound surgeon performance metrics during the prediction of urinary continence recovery after RARP. Nonetheless, many surgeon factors are still independent predictors of early continence recovery. PATIENT SUMMARY: Both patient factors and surgeon kinematic metrics, recorded during robotic prostatectomies, impact early urinary continence recovery after robot-assisted radical prostatectomy.

Development and validation of an objective scoring tool to evaluate surgical dissection: Dissection Assessment for Robotic Technique (DART).

Purpose: Evaluation of surgical competency has important implications for training new surgeons, accreditation, and improving patient outcomes. A method to specifically evaluate dissection performance does not yet exist. This project aimed to design a tool to assess surgical dissection quality. Methods: Delphi method was used to validate structure and content of the dissection evaluation. A multi-institutional and multi-disciplinary panel of 14 expert surgeons systematically evaluated each element of the dissection tool. Ten blinded reviewers evaluated 46 de-identified videos of pelvic lymph node and seminal vesicle dissections during the robot-assisted radical prostatectomy. Inter-rater variability was calculated using prevalence-adjusted and bias-adjusted kappa. The area under the curve from receiver operating characteristic curve was used to assess discrimination power for overall DART scores as well as domains in discriminating trainees (≤100 robotic cases) from experts (>100). Results: Four rounds of Delphi method achieved language and content validity in 27/28 elements. Use of 3- or 5-point scale remained contested; thus, both scales were evaluated during validation. The 3-point scale showed improved kappa for each domain. Experts demonstrated significantly greater total scores on both scales (3-point, p< 0.001; 5-point, p< 0.001). The ability to distinguish experience was equivalent for total score on both scales (3-point AUC= 0.92, CI 0.82-1.00, 5-point AUC= 0.92, CI 0.83-1.00). Conclusions: We present the development and validation of Dissection Assessment for Robotic Technique (DART), an objective and reproducible 3-point surgical assessment to evaluate tissue dissection. DART can effectively differentiate levels of surgeon experience and can be used in multiple surgical steps.

Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery.

BACKGROUND: Our previous work classified a taxonomy of suturing gestures during a vesicourethral anastomosis of robotic radical prostatectomy in association with tissue tears and patient outcomes. Herein, we train deep learning-based computer vision to automate the identification and classification of suturing gestures for needle driving attempts. METHODS: Using two independent raters, we manually annotated live suturing video clips to label timepoints and gestures. Identification (2,395 videos) and classification (511 videos) datasets were compiled to train computer vision models to produce 2- and 5-class label predictions, respectively. Networks were trained on inputs of raw red/blue/green pixels as well as optical flow for each frame. Each model was trained on 80/20 train/test splits. RESULTS: In this study, all models were able to reliably predict either the presence of a gesture (identification, area under the curve: 0.88) as well as the type of gesture (classification, area under the curve: 0.87) at significantly above chance levels. For both gesture identification and classification datasets, we observed no effect of recurrent classification model choice (long short-term memory unit versus convolutional long short-term memory unit) on performance. CONCLUSION: Our results demonstrate computer vision's ability to recognize features that not only can identify the action of suturing but also distinguish between different classifications of suturing gestures. This demonstrates the potential to utilize deep learning computer vision toward future automation of surgical skill assessment.

Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience.

Automated performance metrics objectively measure surgeon performance during a robot-assisted radical prostatectomy. Machine learning has demonstrated that automated performance metrics, especially during the vesico-urethral anastomosis of the robot-assisted radical prostatectomy, are predictive of long-term outcomes such as continence recovery time. This study focuses on automated performance metrics during the vesico-urethral anastomosis, specifically on stitch versus sub-stitch levels, to distinguish surgeon experience. During the vesico-urethral anastomosis, automated performance metrics, recorded by a systems data recorder (Intuitive Surgical, Sunnyvale, CA, USA), were reported for each overall stitch (Ctotal) and its individual components: needle handling/targeting (C1), needle driving (C2), and suture cinching (C3) (Fig 1, A). These metrics were organized into three datasets (GlobalSet [whole stitch], RowSet [independent sub-stitches], and ColumnSet [associated sub-stitches] (Fig 1, B) and applied to three machine learning models (AdaBoost, gradient boosting, and random forest) to solve two classifications tasks: experts (≥100 cases) versus novices (<100 cases) and ordinary experts (≥100 and <2,000 cases) versus super experts (≥2,000 cases). Classification accuracy was determined using analysis of variance. Input features were evaluated through a Jaccard index. From 68 vesico-urethral anastomoses, we analyzed 1,570 stitches broken down into 4,708 sub-stitches. For both classification tasks, ColumnSet best distinguished experts (n = 8) versus novices (n = 9) and ordinary experts (n = 5) versus super experts (n = 3) at an accuracy of 0.774 and 0.844, respectively. Feature ranking highlighted Endowrist articulation and needle handling/targeting as most important in classification. Surgeon performance measured by automated performance metrics on a granular sub-stitch level more accurately distinguishes expertise when compared with summary automated performance metrics over whole stitches.